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SCALE FOR THE MEASUREMENT OF QUALITY 
IN ENGLISH COMPOSITION BY YOUNG PEOPLE 

Section i. Introduction 

Every attempt to measure the efficiency of instruction in a 
school system or to evaluate different methods of educational 
procedure serves to emphasize the importance of standards. 
Proper standards would make it possible to compare with cer- 
tainty the work done in one school or system of schools with 
that done elsewhere. They would make it more difficult for 
mere opinion to control so much of our school-room practice. 
After examining the arithmetical abilities of many pupils in dif- 
ferent representative school systems, Dr. C. W. Stone ('08) 
says : " Probably the truest single expression of the findings of 
this study is summed up in the word diversity. . . . Free- 
dom and initiative are here seen to have led educational prac- 
tice in widely varying paths. Certain paths are those of legiti- 
mate differentiation, but others are waste. . . . The great- 
est need shown by this research is standard of achievement. 
That the great variability herein shown would exist if school 
authorities possessed adequate means of measuring products is 
inconceivable, . . . ."'■ 

Standards are greatly needed in order that we may define 
educational requirements. Results that are accepted in one col- 
lege or school as worthy of a mark of " good," " seventy-five," 
or any other of the common conventional ratings, may or may 
not receive the same credit in another educational system. Such 
differences in the acceptance of results are seldom warranted 
by the variations in the statements of requirements issued by the 
institutions concerned. Indeed, as wide divergence in the ac- 
ceptance of results often exists between schools of the same 

' Arithmetical Abilities and Some Factors Determining Them, Colum- 
bia University Contributions to Education, Teachers College Series, No. 
19- 

I 



2 Measurement of Quality in English Composition 

system, and not infrequently it is found between different de- 
partments in the same institution. 

If there were standards or scales for the measurements of 
results in the various school subjects that would approximate 
the accuracy of the scales used in measuring extension, weight 
and time, educational administrators and investigators would be 
able to measure and express the ef55ciency of a school system in 
terms that would carry conviction. Such standards would also 
make it possible to define exactly the requirements of the col- 
lege, high school, and civil service: and by them school superin- 
tendents could define requirements for promotions from one 
grade to the next. 

The more or less satisfactory standards or scales that have 
been established in certain school subjects show that the task, 
while difficult, is by no means impossible. Dr. J. M. Rice 
('97)" tested the spelling ability of some 3,000 public school 
pupils in 21 different school systems. His purpose was to com- 
pare the conditions existing in the various schools and to de- 
termine which influenced the spelling ability of the pupils. In 
order to do this he developed a series of tests which were given 
in each of the schools under as nearly the same conditions as 
possible. When the returns from these tests were scored, the 
tabulated results furnished a standard with which any teacher 
might compare the work of a class in spelling. 

Dr. O. P. Cornman ('02)* made a more intensive study of the 
same character in several of the Philadelphia schools. He used 
some of the Rice tests and also developed others. The tests 
were uniformly scored and the results were tabulated in proper 
form for making comparisons between the various schools. 

Dr. Rice ('02)* also tested the arithmetical abilities of about 
6,000 school children in 17 schools in 7 cities. The methods 
employed in this research were similar to those employed in the 
spelling investigation. 

'The Futility of the Spelling Grind, The Forum, Vol. 23, pp. 163-172 
and 409-419. 

'Spelling in the Elementary School; An Experimental and Statistical 
Investigation. 

* Educational Research : A Test in Arithmetic, The Forum, Vol. 34, 
pp. 281-297. Causes of Success and Failure in Arithmetic, The Forum, 
Vol. 34. PP- 437-452. 
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Dr. Stone ('08)' tested the arithmetical abilities in funda- 
mentals and reasoning of some 6,000 school children in the 6A 
(high 6th) grade in 26 representative public school systems. 
Two of the problems on which this study has specific bearing 
are: "(i) What is the nature of the product of the first six 
years of arithmetic work? (2) What is the relation between 
distinctive procedures in arithmetic work and the resulting 
abilities?" A carefully differentiated test in the fundamentals 
was used and a separate test was given for reasoning abilities. 
As was done in all the other investigations, the results were 
uniformly scored, only in this case greater effort was made to 
evaluate the different problems. The results are to be found 
in tabular form. Dr. Stone says of his work, " It is believed 
that the present study will help to standardize the work in 
arithmetic in the first six grades. Anyone who wishes may 
know how his system or school compares with the representative 
systems of the country." The author gives explicit directions 
for the use of the tests, the method of scoring the results, and 
the use of the tables in comparing results. Subsequently these 
tests have been tried in a number of places, and the results 
have served to indicate the relative efficiency of the different 
schools. 

The whole process of developing such standards is well shown 
in the work which Mr. S. A. Courtis ('11)* is conducting. He 
has developed an improved set of arithmetic tests by which it is 
claimed any grade in the school may be tested. These have been 
given to many thousands of children under uniform conditions. 
The results, -when properly tabulated, will form a scale by which 
it will be possible to measure the achievement of any class or 
pupil. 

The feature of the studies which makes their consideration 
relevant to this study is not the conclusions reached but the 
fact that in each case tests were given to a sufficient number of 



' Arithmetical Abilities and Some Factors Determining Them, Columbia 
University Contributions to Education, Teachers College Series, No. 19. 

'Measurements of Growth and Efficiency in Arithmetic, Elementary 
School Teacher, Vol. 10, pp. 58-74 and 177-199. Elementary School 
Teacher, Vol. 12, pp. 127-137. Journal Educational Psychology, Vol. 2, 
p. 272. The Courtis Standard Tests in Arithmetic, with Manual of 
Instructions for giving and scoring. 



4 Measurement of Quality in English Composition 

pupils to enable the authors to establish a standard or scale with 
which they were able to measure the relative efficiency of the 
various systems. 

Dr. E. L. Thorndike ('lo)' has published a scale for the 
measurement of quality in the handwriting of children and also 
one for the handwriting of women. The derivation of these 
scales involved a somewhat different procedure from that em- 
ployed for the scales in spelling and arithmetic. Many sam- 
ples both of children's handwriting and of adults' were secured 
and these were judged separately by many competent indi- 
viduals. From the scores thus obtained the relative values of 
thfe samples were determined and the scales formed. The 
scales in this case consist of a series of samples of hand- 
writing with which the samples that are to be measured are 
compared. In many particulars the methods employed in de- 
riving these scales in handwriting are the same as those used in 
the present study, and frequent reference will be made to Dr. 
Thorndike's study in the descriptions which follow. 

'Handwriting, Teachers College Record, Vol. ii, No. 2 [reprinted]. 



Section 2. The Scale 

This scale for the measurement of quality in English compo- 
sition by young people has been derived by methods that are 
described in the following pages. 

Sample 580. Value 0. Artificial sample. 

Letter. 
Dear Sir: I write to say that it aint a square deal Schools 
is I say they is I went to a school, red and gree green and 
brown aint it hito bit I say he don't know his business not to- 
day nor yeaterday and you know it and I want Jennie to get 
me out. 

Sample 595. Value 183. Artificial sample. 

My Favorite Book, 
the book I refer to read is Ichabod Crane, it is an grate book 
and I like to rede it. Ichabod Crame was a man and a man 
wrote a book and it is called Ichabod Crane i like it because the 
man called it ichabod crane when I read it for it is such a great 
book. 

Sample 618. Value 260. Artificial sample. 

The Advantage of Tyranny. 
Advantage evils are things of tyranny and there are many ad- 
vantage evils. One thing it that when they opress the people 
they suffer awful I think it is a terrible thing when they say 
that you can be hanged down or trodden down without mercy 
and the tyranny does what they want there was tyrans in the 
revolutionary war and so they throwed off the yok. 

Sample 94. Value 369. Written by a boy in the second year 

of the high school, aged 14 years. 

Sulla as a Tyrant. 

When Sulla came back from his conquest Marius had put 

himself consul so sulla with the army he had with him in his 

conquest siezed the government from Marius and put himself in 

consul and had a list of his enemys printy and the men whoes 

names were on this list we beheaded. 

5 



The Scale 7 

Sample 519. Value 474. Written by a girl in the third year 
of the high school, aged 17 years. 

De Quincy. 

First: De Quincys mother was a beautiful women and 
through her De Quincy inhereted much of his genius. 

His running away from school enfluenced him much as he 
roamed through the woods, valleys and his mind became very 
meditative. 

The greatest enfluence of De Quincy's life was the opium 
habit. If it was not for this habit it is doubtful whether we 
would now be reading his writings. 

His companions during his college course and even before 
that time were great enfluences. The surroundings of De 
Quincy were enfluences. Not only De Quincy's habit of opium 
but other habits which were peculiar to his life. 

His marriage to the woman which he did not especially care 
for. 

The many well educated and noteworthy friends of De 
Quincy. 



Sample 534. Value 585. Written by a boy in the fourth year 
of the high school, aged 16 years. 

Fluellen. 

The passages given show the following characteristic of 
Fluellen: his inclination to brag, his professed knowledge of 
History, his complaining character, his great patriotism, pride 
of his leader, admired honesty, revengeful, love of fun and 
punishment of those who deserve it. 



Sample 196. Value 675. Written by a girl in the first year 
of the high school, aged 18 years. 

Ichabod Crane. 

Ichabod Crane was a schoolmaster in a place called Sleepy 
Hollow. He was tall and slim with broad shoulders, long arms 
that dangled far below his coat sleeves. His feet looked as if 
they might easily have been used for shovels. His nose was 
long and his entire frame was most loosely hung to-gether. 
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Sample 221. Value 772. Written by a boy in the third year of 
the high school, aged 16 years. 

Going Down with Victory. 

As we road down Lombard Street, we saw flags waving from 
nearly every window. I surely felt proud that day to be the 
driver of the gaily decorated coach. Again and again we were 
cheered as we drove slowly to the postmasters, to await the 
coming of his majestie's mail. There wasn't one of the gaily 
bedecked coaches that could have compared with ours, in my 
estimation. So with waving flags and fluttering hearts we 
waited for the coming of the mail and the expected tidings of 
victory. 

When at last it did arrive the postmaster began to quickly 
sort the bundles, we waited anxiously. Immediately upon re- 
ceiving our bundles, I lashed the horses and they responded with 
a jump. Out into the country we drove at reckless speed — 
everywhere spreading like wildfire the news, "Victory !" The ex- 
ileration that we all felt was shared with the horses. Up and down 
grade and over bridges, we drove at breakneck speed and 
spreading_ the news at every hamlet with that one cry " Vic- 
tory ! " When at last we were back home again, it was with 
the hope that we should have another ride some day with " Vic- 
tory." 



Sample 571. Value 838. Written by a boy in the Freshman 
class in college. 

Venus of Melos. 

In looking at this statue we think, not of wisdom, or power, 
or force, but just of beauty. She stands resting the weight 
of her body on one foot, and advancing the other (left) with 
knee bent. The posture causes the figure to sway slightly to 
one side, describing a fine curved line. The lower limbs are 
draped but the upper part of the body is uncovered. (The un- 
fortunate loss of the statue's arms prevents a positive knowl- 
edge of its original attitude.) The eyes are partly closed, hav- 
ing something of a dreamy languor. The nose is perfectly cut, 
the mouth and chin are moulded in adorable curves. Yet to say 
that every feature is of faultless perfection is but cold praise. 
No analysis can convey the sense of her peerless beauty. 



The Scale ii 

Sample 177. Value 937. Written by a boy in the Freshman 
class in college. 

A Foreigner's Tribute to Joan of Arc. 

Joan of Arc, worn out by the suffering that was thrust upon 
her, nethertheless appeared with a brave mien before the Bishop 
of Beauvais. She knew, had always known that she must die 
when her mission was fulfilled and death held no terrors for 
her. To all the bishop's questions she answered firmly and with- 
out hesitation. The bishop failed to confuse her and at last 
condemned her to death for heresy, bidding her recant if she 
would live. She refused and was lead to prison, from there to 
death. 

While the flames were writhing around her she bade the old 
bishop who stood by her to move away or he would be injured. 
Her last thought was of others and De Quincy says, that recant 
was no more in her mind than on her lips. She died as she 
lived, with a prayer on her lips and listening to the voices that 
had whispered to her so often. 

The heroism of Joan of Arc was wonderful. We do not 
know what form her great patriotism took or how far it really 
led her. She spoke of hearing voices and of seeing visions. We 
only know that she resolved to save her country, knowing 
though she did so, it would cost her her life. Yet she never 
hesitated. She was uneducated save for the lessons taught her 
by nature. Yet she led armies and crowned the dauphin, king 
of France. She was only a girl, yet she could silence a great 
bishop by words that came from her heart and from her faith. 
She was only a woman, yet she could die as bravely as any 
martyr who had gone before. 



Section 3. What the Scale Measures 

The scale is composed of sample English compositions, the 
qualities of which have been determined by more than four 
hundred competent judges. The values which are assigned to 
the various samples express their quality in the same sense, 
though not as accurately, as millimeters express the lengths of 
lines. Just as 183 mm. may be expressed by 18.3 cm. or 1.83 
dm., so the 183 employed in this scale may be considered as 
183 small units of quality, or as 18.3 units ten times as large, 
or as 1.83 units one hundred times as large. The values here 
employed should not be confused with the customary per cents 
that are used in rating English compositions. A difference of 
two hundred in this scale is equal to twice a difference of one 
hundred taken in any part of the scale. Thus sample 94 with 
a value of 369 is a little more than twice as good an English 
composition as sample 595 with a value of 183. Sample 519 
with a value of 474 is about one-half as good as sample 177 with 
a value of 937. 

Mer it in English writing is complex. Judges are influenced 
both by form and by content. Such factors of form as spell- 
ing, punctuation, capitalization, and the like are subject to defi- 
nite rules. Form is, therefore, more easily measured than con- 
tent. When an individual is in doubt concerning the relative 
merits of two English compositions, the tendency is to fix upon 
one or more of the obvious form elements and for the time 
being to give them undue importance in fixing the relation of 
the samples. 

No attempt has been made in this study to define merit. 
The term as here used means just that quality which competent 
persons commonly consider as merit, and the scale measures 
just this quality. The accompanying graphic representation of 
the location of the various samples composing the scale will 
serve to make plain the meaning of the scale. 

The value -ij f any English composition may be obtained 
by placing it beside the samples constituting the scale and 
determining to which it most nearly corresponds. The 
use of the scale will be clearly understood, if the reader 

13 



14 Measurement of Quality in English Composition 

will compare sample 376, given below, with the scale. 
There will be no hesitation in pronouncing sample 376 
better than the lower end of the scale until quality 585 
is reached. The majority of persons will consider sample 376 



a m jut je9 . 



SSO S9S tIS H SI9 Si^ lit ZV J7f 111 

Fig. I. _ Graphic representation of the relative values of the samples com- 
posing the scale. The numbers above the line represent the various 
unit points in the scale. The numbers below the line represent sample 
compositions and show their relative positions on the scale. Thus, 
sample 580 is at the zero point, sample S9S is between the values 
of 100 and 200, etc. 



worth about 772 or the equal of sample 221. In an actual test 
with seventy-three individuals, seventeen considered sample 376 
inferior to sample 534, sixteen thought it inferior to sample 196, 
and twenty-one thought sample 376 was better than sample 571 ; 
but there was almost perfect agreement that sample 376 was 
nearly if not quite equal to sample 221. If the reader thinks 
sample 376 is better than sample 221 but not as good as sample 
571, he may place the value between 772 and 838. By this 
method the value of any sample may be expressed as accurately 
as the individual cares to make it. " The sample to be measured 
should, for convenience, be examined with the entire scale in 
view. If the scale's samples are arranged in order on a table or 
against a wall, the examined sample is easily compared with them. 
The measurer then decides what quality of the scale the sample 
possesses and records the measure. . . . The measure may 
be made more and more accurate by having othei; judges also 
measure, each always in ignorance of the ratings given by the 
others. In default of other judges, the measure may be made 
more accurate by rating the sample two or more times, each 
time in ignorance of the ratings previously given. An individual 
may be measured more accurately by using several samples of 
his compositions, each being rated in ignorance of the ratings 
given to the other samples."* 



' Handwriting, p. 8. 
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Sample 376. The Military Career of Alexander the Great. 

Alexander the Great one of the greatest generals of the 
world began his military career at the age of twenty. It was 
then at his father's death that he crushed all rebellion both in 
Macedonia and Greece. Then he started on his famous march. 
Crossing the Hellespont he defeated the Persians at the river 
Granicus from thence he followed the coast of Asia Minor. 
However he turned aside from his course and going up into 
Phrygium he cut the Gordian knot. Next he defeated a large 
Persian army at Issus, captured Tyre after a long siege and 
much labor. He now went into Egypt, founded Alexandria, 
and returned to Persia. At Arbela the Persian army was an- 
nihilated and Darius fled. Alexander next conquered Babylon 
and Persepolis and then having wandered thru eastern Persia 
he crossed the Hindu Hush Mountains and entered India. Al- 
though victorious Alexander's army was weary of such hard 
travelling and refused to go farther. So the great general was 
forced to lead them back to Babylon and his military career was 
finished. 



Section 4. The Location of the Zero Point in English 

Composition' 

The location of the zero point for merit in English composi- 
tion by young people is of importance in order to allow the 
" times as much merit " judgment. Measures of fatigue, prac- 
tice, or change of any sort will be greatly facilitated if we can 
so arrange the scale that 8, 10, and 12 on it mean twice as far 
from just not any of the thing in question as 4, 5, and 6; 
that 240 means twice as much of the thing as 120, three times 
as much as 80, five times as much as 48. Hitherto no one could 
say with any assurance whether sample 571 was two, ten or a 
hundred times as " good " as 618. 

fit has been located approximately as sample 580 on the basis 
of the judgments of (i) nine men of special literary ability, five 
/of them professors of English, of whom four are also authors 
' of standard text-books on English composition, and four 
men of marked general intellect and literary ability, (2) 
eleven gifted teachers familiar with secondary education, and 
(3) eight psychologists familiar with the significance of scales 
and zero points in the case of intellectual abilities and products. 

Although no one person of these had any deliberate criterion 
for the point where positive merit just begins, and although 
one's first reaction to the request to locate such a point is to 
regard it as arbitrary, there is much agreement among indi- 
viduals and almost perfect agreement in the case of the averages 
of the three groups. 

These individuals located the zero point in connection with 
the set of 27 samples*" either by placing it below the worst of 
these or by stating which of these they considered to possess ab- 
solutely no merit as specimens of English composition by young 
people. 

Of the twenty-eight judges, two regard zero merit as some- 
thing below the least meritorious specimens of this list, and 
five put it higher than sample 618, but three-fourths of the 

_ " For the experimentation and calculation in connection with the loca- 
tion of the zero point. Professor E. L. Thomdike is responsible. 
"For the samples used in this work see Section 8. 

16 
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judges locate it as lower than sample 607 or higher than sample 
595. The central tendency of the twenty-eight puts it at or just 
barely below specimen 580. 

Taking the sample regarded as just barely of some merit and 
that regarded as the next below it in the case of each judge, we 
have the following results: 



Below 607- 


4 times 


Below 580 


5 times 


" 607 


9 " 


Cl 


39 


3 " 


" 670- 


2 . " 


H 


79 


6 " 


58 


3 " 


U 


595 


2 " 


" 501. 


6 « 


U 


603 


2 " 


" 491 


3 " 


11 


618 


2 " 






Above 618 


9 '' 



The median opinion of these 28 judges thus places the begin- 
ning of merit in English writing by young people at sample 580. 



Section 5. Theory of the Method Employed in Deriving 

THE Scale 

Nearly five hundred individuals judged one or the other of 
the sets which contained the samples used in the scale. Each 
person was asked to arrange the samples in order of merit. 
This amounted to a request that each sample should be com- 
pared with every other sample in the set. In scoring the results, 
the poorest sample was numbered one, the next poorest two, and 
so on; thus in a set of eighty-three samples the best was num7 
bered eighty-three. The scale was derived from these scores by 
the method of right and wrong cases.^^ The theory of this 
method as applied to this study may be stated as follows : Dif- 
ferences that are equally often noticed are equal, unless the 
differences are either always or never noticed.^'' 

Merit in English writing is the resultant of a large number of 
independent factors. Among the individuals who judged the 
samples used in this study, several greatly undervalued certain 
samples because they thought that the language used was too 
mature for high school pupils. On the other hand, several of 
the artificial samples were greatly overvalued because the judges 
thought that they detected childish modes of expression in 
these. When a large number of individuals are required to 
judge the quality of any sample English composition, many will 
be nearly correct, some will be in error because of overvaluing it, 
and an equal number will undervalue it. Small errors of judg- 
ment will be much more common than large ones. The theory of 
probability enables us to construct a diagram that will represent 
what would result if a large number of persons were to judge 
an English composition, or what would result if one person 
were to judge independently a large number of times. Figure 

11 shows the probable grouping of judgments, when a large 
number of individuals are requested to determine the value of 
a composition. The line XO represents diagrammatically the 
value of an English composition. The bell-shaped surface, 
ABC, shows the distribution of judgments. The shape of the 

"Fullerton and Cattel, On the Perception of Small Differences, pp. 

12 ff. 

*" Handwriting, p. 5. 

18 
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curve will depend upon the accuracy of the judges. The better 
the judges the higher that part of the curve above O and the 
shorter the base AC. The surface ABC represents diagram- 
matically a normal surface of distribution. The line BO divides 
the surface into two equal parts. PQ and MN are equally 
distant from BO, and within the area PMNQ, fifty per cent 
of the judgments are found.^* The line OP or its equal OM 
represents an error of such size that half of the errors are larger 
and half are smaller. It is the median error or as it is called 
the median deviation (M. D.)^* 




Fig. II. A normal surface of frequency on which is shown the distri- 
bution of errors in judgment when a large number of_ individuals are 
required to determine the value of a sample English composition 
whose value by consensus of opinion is represented by the line XO. 



In Fig. Ill, the values of two samples are represented, one 
by the line XO and the other by the line X'O'. Such a difference 
in value is taken that the line O'P', which represents this dif- 
ference, is equal to the median deviation. The area P'C'Q' 
contains seventy-five per cent of the judgments of the sample 
represented by X'O'. Seventy-five per cent, therefore, of the 
judgments are correct and twenty-five per cent (the area 
A'P'Q') are wrong, because they make the sample represented 
by XO better than the sample represented by X'O'. When 
seventy-five per cent of the judges notice the difference in 
quality between two samples, the difference is equal to the 
median deviation, and the ratio between the difference and the 
median deviation in this case is i/i or i. Throughout this 
study this ratio is taken as the unit of value. The unit may be 



"For a complete statement of the relations that exist between the 
various parts of a normal surface of distribution, see Thorndike, Mental 
and Social Measurements, p. Sp. 

"The name 'probable error' is applied to this value. 
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defined as that difference in quality which exactly seventy-five 
per cent of the judges observe. 

In Figs. IV and V the arrangement of the curves is such as 
to show the relation that exists between the difference and the 
median deviation when various percentages of the judges notice 
the difference. Curves No. i and No. 2 show this relation when 




Fig. III. Two surfaces of frequencies so placed as to show the difference 
in value of two samples of English compositions when seventy-five 
per cent of the judges consider one better than the other. 

fifty per cent of the judges believe that sample represented by 
curve No. 2 is better than sample represented by curve No. i. 
In this case BO extended passes through point O' and there 
is no difference in the quality of these samples. Curve No. 3 
is so placed that fifty-five per cent of the judges consider sample 
represented by curve No. 3 better than sample represented by 
curve No. i. The difference is represented by the line Y'O". 
By actual measurement Y'O" is tVj of P"0". Curve No. 4 
shows sixty per cent "better" judgments. The difference 
Y"'0"' is T^ of the median deviation (P"'0"'). In curve No. 
5 the percentage of "better" judgments is sixty-five and the 
difference is iWr of the median deviation. The following table 
shows the curve compared with curve No. i, the percentage of 
better judgments represented by the curve, and the ratio between 
the difference in value and the median deviation ( P- ). Median 
deviation is defined on page 19. 
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TABLE I 

[■he Ratios Between the Dipfebences in Value and the Median Devia- 
tion COKRESPONDING TO CERTAIN PERCENTAGES OF "BeTTBr" 

Judgments as Shown by the Curves in Figs. Ill, IV and V 



Curve compared 
with curve No. 1 



Percentage 
of "better" 
judgments 



Difference divided by 

the median deviation 

(D/M. D.) 



Fig. IV 

U 
U 
tl 
U 

Fig. Ill 
Fig. V 



2 
3 
4 
5 



No. 
No. 
No. 
No. 
No. 6 
No. 2 
No. 7 
No. 8 
No. 9 
No. 10 
No. 11 



60 
55 
60 
65 
70 
75 
80 
85 
90 
95 
100 





.19 

.38 

57 

.78 

1.00 

1.25 

1.53 

1.89 

2.43 

7 



A table giving the differences for all percentages from fifty 
to ninety-nine is found on page i6 of " On the Perception of 
Small Differences." This table is also found on page 164 of 
" Mental and Social Measurements." In selecting the samples 
that should be used in the scale, it was more convenient to begin 
with the poorer samples, so the table was transmuted to give 
the differences for percentages from i to 50, and the differences 
for tenths of a per cent were interpolated. The results are 
found in Table II. 

Any standard or scale should be based on an unit such that 
equal units may be derived independently of the scale. The 
unit in this scale has been defined as that difference which 
seventy-five per cent of the judges are able to distinguish. All 
that is required to derive this unit is a set of samples that vary 
from each other by small degrees in quality. When two sam- 
ples are found such that seventy-five per cent of the judges , 
agree in calling one better than the other, the difference is just 
the difference used as the unit in this scale. 
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B 




No. 1. 



No. 2. 



No. 3. 



No. 4. 



No. 5. 



No. 6. 



pv- 0" M" 



Fig. IV. Various frequency curves so placed in relation to curve number 
one as to show the difference in value between two samples when 
the percentage of "better" judgments is either So, 55, 6o, 65, or 70 
respectively. 
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No. 1. 



No. 7. 



No. 8. 



No. 9. 



No. 10. 



No. 11. 




Fig. V. Various frequency curves so placed in relation to curve number 
one as to show the difference in value between two samples when 
the percentage of " better " judgment is either 80, 85, 90, 95, or 100 
respectively. 
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TABLE II 

Table for Determining the Median Deviation teom the Percentage 
OF Wrong Cases and the Amount op Difference 

The table reads as follows r When i ^ per cent of the judges regard 
sample A worse than sample. B, the difference, A-B, is 3.45 of the median 
deviation. "When i.i per cent of the judges regard sample A worse 
than sample B, the difference, A-B, is 3.41 of the median deviation. When 
1.2 per cent of the judges regard sample A worse than sample B, the 
difference, A-B, is 3.37 of the median deviation, etc. 



a=«.s 




fsl 




a=„js 




a=„3 




a=«l 




III 


D 


IS^S 


D 


III 


D 




D 


lif 


D 


P«-S-" 


M. D. 


M. D. 


M. D. 


M. D. 


M. D. 


1.0 


3.45 


6.0 


2.31 


11.0 


1.82 


16.0 


1.47 


21.0 


1.20 


.1 


3.41 


.1 


2.30 


.1 




81 


.1 


1.46 


.1 


1.19 


.2 


3.37 


.2 


2.29 


.2 




80 
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1.46 


.2 


1.19 
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3.33 
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3.29 


.4 


2.26 
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.6 


.97 


7 


2.35 


.7 


1.84 
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.7 
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1.83 
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1.83 


.9 
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Section 6. The Sample Compositions from Which the 
Scale was Derived 

The first step in the derivation of the scale was the collec- 
tion of about seven thousand English compositions by young 
people. These were obtained from various sources and repre- 
sent a definite attempt to obtain particularly the very poorest 
and the best work that is done in the schools. After collect- 
ing these samples the author and one other graded them roughly 
into ten classes. From these classes seventy-five samples were 
selected. In order that the samples at the extremes of the scale 
might be measured, it was necessary to supply some artificial 
samples. The poorest of these were conscious attempts by adults 
to write very poor English. The best samples were obtained f rom | 
youthful writings of such literary geniuses as Jane Austen and ^ 
the Brontes. The works of some college Freshmen were also 
used. As finally constituted, the set consisted of eighty-three 
samples which varied from the poorest to the best by small 
degrees of quality. '' 

Nearly all of the samples were in the handwriting of the 
authors, some of which was very good and some very poor. 
In order that the character of the handwriting might not in- 
fluence the judges, all the samples were typewritten and mimeo- 
graphed. Great care was exercised in reproducing all of the 
mistakes that appeared in the originals. 

It would have been desirable to have included more than the 
eighty-three samples, but the persons whose services were par- 
ticularly desired were busy, and since the judging of a set of 
even eighty-three required from three to six ..hours of close 
application, it was not possible to enlarge the set. 

Separate sets of these samples were given to about one 
hundred, indixylJiais* with the following printed request : " Please 
arrange these in order of merit as specimens of English com- 
position by young people. If there are two or more of them 
which seem to- you to be absolutely equal in merit, give them 
the same number as a score. Score your results on the accom- 
panying sheet calling the worst specimen i, the next worst 2, 
the next worst 3, and so on." 
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Section 7. The Results Obtained with the First Set 
OF Samples and the Selection of a Second Set 

More than one hundred individuals worked with the set of 
eighty-three samples. Some, however, did not understand the di- 
rections and some made such serious mistakes in recording their 
results that it was possible to use only seventy-three of the 
records. These records were copied on a large sheet for con- 
venience in handling, and the distribution of the positions as- 
signed to each sample was determined. The next step was to 
compare the position assigned each sample with that assigned 
each other sample in order to determine just how many of 
the judges had considered any given sample better than any 
other given sample. These " better " judgments were arranged 
in tabular form such as is shown in Table IV on page 30. 

There were great variations in the positions assigned to the 
samples. It was not possible to establish the position of any 
sample with reasonable accuracy until many more judgments 
could be secured. The judgments of the seventy-three indi- 
viduals indicated the samples that were probably nearly equal 
in merit, and in this way made it possible to select from the 
eighty-three samples a much smaller set which would still con- 
tain all the important steps in quality from the poorest to the 
best. The samples that were included in the new set are re- 
produced in Section 8, and reference to Table V, which shows 
the percentage of " better " judgments for the samples selected 
from the set of eighty-three, will help in understanding the 
method that was employed in selecting this set. 

All but three of the judges had agreed that sample 607 was 
the poorest. This sample was included, as well as the next 
samples in merit, 580 and 595. The remaining eipfht^en sam- 
ples were selected by taking the sample that about seventy-five 
per cent of the judges had agreed was better than , the last 
sample selected. The differences in merit between samples 607 
and 580, and between samples 580 and 595 were large. New 
artificial samples were included with these three samples and 
these were judged by a number of individuals in order to de- 
termine several samples that would range in merit between the 
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two pairs. In this way five were selected, making in all a set 
of twenty-seven samples. Later samples 519 and 520 were 
added. 

Since only twenty-three of the eighty-three samples in the 
first set are used in any succeeding set, only such data as bear 
upon these twenty-three samples are here given. Table III 
shows the distribution of positions assigned to these samples 
by seventy-four individuals. This table is followed by Table IV 
which gives the number of "better" judgments for this set, 
and by Table V which shows the percentage of " better " judg- 
ments for each pair of samples. Only the significant data are 
given in Tables IV and V. 
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Section 8. Results Obtained with the Second Set of 

Samples 

The method employed in selecting the samples that were 
placed in the second set was given in the preceding section. 
The set is reproduced below except that samples 519 and 520 
were not included in all of the sets judged. 

y^ Sample 607. Artificial. 

Sketch. 

I words four and two came go billa guni sing hay cows and 
horses he done it good he died it goon I want yes sir yes sir 
oxes and sheeps he come yes sir camed and goes billum gumun 
oomunn goodum. 

Sample 491. Artificial. 

I say never mind. I say he knows all right all right dony 
puki I say it aint. I got one I got to noo and loo, lov and sov. 
Dont you care all right all right I say 

Sample 58. Artificial. 

i from thre ours up in room on books and books care for chil- 
der tore a page and rite on them 

Sample 501. Artificial. 

dere techer : 

I like schol not like schol. that man other place like make 
work tools. Some day you say I rede 

Sample 670. Artificial. 

I want to say it aint no youse they aint got no right and they 
aint got no man ban tan pan pan san san sem sen sun sun tun 
tun 

y" Sample 580. Artificial. 

Letter. 

Dear Sir: I write to say that it aint a square deal Schools 
is I say they is I went to a school, red and gree green and 

32 
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brown aint it hito bit I say he don't know his business not to- 
day nor yeaterday and you know it and I want Jennie to get 
me out. 



Sample pf. Artificial. 

I write to tell I like the job to work what mr Lambert said 
as he giv me a dolar harf a day to do things round his plase. 
I not affrade to work so I say plees giv me the job and I can 
look after the cow all right or anything no trouble long time I 
work for mr. panter he knows 



Sample 39. Artificial. 

The man take little girls in big to the schul when it rains, the 
wagon with thing to keep warm and light up the rode all the 
way to Mis Gow. 

\/ Sample 595. Artificial. 

My Favorite Book. 

the book I refer to read is Ichabod Crane, it is an grate book 
and I like to rede it. Ichabod Crame was a man and a man 
wrote a book and it is called Ichabod Crane i like it because the 
man called it ichabod crane when I read it for it is such a great 
book. 



Sample 618. Artificial. 

The Advantage of Tyranny. 

Advantage evils are things of tyranny and there are many 
advantage evils. One thing is that when they opress the people 
they suffer awful I think it is a terrible thing when they say 
that you can be hanged down or trodden down without mercy 
and the tyranny does what they want there was tyrans in the 
revolutionary war and so they throwed off the yok. 

Sample 60s. Artificial 

A Character Sketch. 

The rtif-n I am describing is a white man and he has nice 
hair and wears a hat, and his horse is black, I like this man and 
he has two eyes and his nose is red. 
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Sample 570. Artificial. 

Description of School Room. 

Our school room is on the side of the school house and it 
is a awfully nice room and I like it because it is so nice and 
all the boys like it, and there is a good many pictures on the 
wall and there is a clock on the wall. We like this school room 
and come to school most all the time. 



Sample 627. Artificial. 

A Scene. 

I think the sunlight is very beautiful on the water, and when 
it shines on the water it is very beautiful, and I love to watch 
it when it is so beautiful. The colors are so pretty and the noise 
of the water with the sunshine are so attractive in the sunshine 
I wonder do other people love to watch the water like I do. 
I dont know as there is anything as lovely as the water waves in 
the sunlight of the glorious orb. 



Sample P4. 

Sulla as a Tyrant. 

When Sulla came back from his conquest Marius had put 
himself consul so sulla with the army he had with him in his 
conquest siezed the government from Marius and put himself 
in consul and had a list of his enemys printy and the men whoes 
names were on this list we beheaded. 



Sample zoo. 

My dear Fred, — 

I will tell you of my journey to Delphi Falls, N. Y. There 
is nice scenery along this route. The prettiest scene is in the 
glulf which is quite narrow, a small creek flows down it and 
the road follows along near its banks. 

There are woods on either side, these trees look very pretty 
when they are white with snow. 

In summer it is always shady and cool in them and the small 
fish may be seen darting back and forth in the water. 

I hope I will have the pleasure of taking you over the route 
some time. Yours sincerely. 
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Sample 322. 

A Picture. 

I should like to see a picture; illustrating a part of L'allegro. 
Where the godesses of Mirth and Liberty trip along hand in 
hand. Two beautiful girls dressed in flowing garments, danc- 
ing along a flower-strewn path, through a pretty garden. Their 
hair flowing down in long curls. Their contenances showing 
their perfect freedom and happiness. Their arms extended 
gracefully smelling some sweet flower. In my mind this would 
make a beautiful picture. 

Sample 534, 

Fluellen. 

The passages given show the following characteristic of Flu- 
ellen : his inclination to brag, his professed knowledge of His- 
tory, his complaining character, his great patriotism, pride 
of his leader, admired honesty, revengeful, love of fun and 
punishment of those who deserve it. 

i 

Sample 196. 

Ichabod Crane. 

Ichabod Crane was a schoolmaster in a place called Sleepy 
Hollow. He was tall and slim with broad shoulders, long arms 
that dangled far below his coat sleeves. His feet looked as 
if they might easily have been used for shovels. His nose was 
long and his entire frame was most loosely hung to-gether. 

Sample 221. 

Going Down with Victory. 

As we road down Lombard Street, we saw flags waving from 
nearly every window. I surely felt proud that day to be the 
driver of the gaily decorated coach. Again and again we were 
cheered as we drove slowly to the postmasters, to await the 
coming of his rnajestie's mail. There wasn't one of the gaily 
bedecked coaches that could have compared with ours, in my 
estimation. So with waving flags and fluttering hearts we 
waited for the coming of the mail and the expected tidings of 
victory. 

When at last it did arrive the postmaster began to quickly 
sort the bundles, we waited anxiously. Immediately upon re- 
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ceiving our bundles, I lashed the horses and they responded 
with a jump. Out into the country we drove at reckless speed 
— everywhere spreading like wildfire the news, " Victory !" The 
exileration that we all felt was shared with the horses. Up 
and down grade and over bridges, we drove at breakneck speed 
and spreading the news at every hamlet with that one cry " Vic- 
tory ! " When at last we were back home again, it was with 
the hope that we should have another ride some day with " Vic- 
tory." 



Sample 300. 

The Preacher of Auburn. 

The most popular man of Auburn was the preacher. Al- 
though he had a very small salary he was contented. The 
preacher was kind to everybody. Little children loved him. Old 
soldiers liked to sit by his fireside and tell stories of the battles, 
which they had fought in. The beggars who came to his door, 
although chided for leading such an existence, were always 
clothed and feed. 

The preacher was always willing to go to the homes where 
there was sickness or death. Here he helped in all things that 
he could. 

In the church he preached with unaffected grace, and all who 
came to scoff at him remained to worship. 

The minister was a contented, simple and kind man, whom 
the people loved. 

Sample 525. Artificial. 

Essay on Bums. 

As far as I can learn from the Essay on Burns, Mr. Carlyle 
considers that good poetry must contain the sincerity of the 
poet. The poem must show the author's good choice of sub- 
ject and his clearness of sight. In order to have good poetry 
the poet must be familiar with his subject and his poem will 
show it. 

The characteristics of a great poet, in Mr. Carlyle's opinion 
were sincerity and choice of subject. A poet must be appre- 
ciative of nature and have a responding heart. Carlyle says a 
true poet does no have to write on subjects which are far away 
and probably come from the clouds. A truly great poet makes 
the most of subjects which are familiar to him and close to 
earth, as Burns did in his poems to the Field Mouse and The 
Daisey. 
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Sample ^65. Artificial. 

October. 

I can say without blushing that I think I was born in the best 
month of the year — October. There is something in a fair 
October day that makes me feel healthier and happier than any 
other time of the year. I think there are good reasons for this 
feeling. 

At what time of the year do we have such crisp mornings, 
such glorious noons and such soft cool twilights? All ttie 
sweetness of summer seems to be merging into the chill of 
winter. There is contentment in the atmosphere. The fanner 
is rejoicing over his crops ; the hunter is seeking satisfaction and 
pleasure in the forest; the business man is looking forward to 
the holiday trade; the college man, fresh from his vacation, is 
at his best ; and Nature herself is aflame with color. The whole 
world seems to be at peace with itself. It is the vacation time 
for Mother Earth, a lull between the torrid heat of summer and 
the cold blasts of winter. 

Sample 424- Artificial. 

A Diary. 

I had an early run in the woods before the dew was off the 
grass. The moss was like velvet and as I ran under the arches 
of yellow and red leaves I sang for joy, my heart was so bright 
and the world was so beautiful. I stopped at the end of the 
walk and saw the sunshine out over the wide " Virginia 
meadows." 

It seemed like going through a dark life or grave into heaven 
beyond. A very strange and solemn feeling came over me as 
I stood there, with no sound but the rustle of the pines, no one 
near me, and the sun so glorious, as for me alone. It seemed 
as if -I had felt God as I never did before, and I prayed in my 
heart that I might keep that happy sense of nearness all my 
life. 

Sample ilf. Artificial. 

A Foreigner's Tribute to Joan of Arc. 

Joan of Arc, worn out by the suffering that was thrust upon 
her, nevertheless appeared with a brave mien before the Bishop 
of Beauvais. She knew, had always known that she must die 
when her mission was fulfilled and death held no terrors for 
her. To all the bishop's questions she answered firmly and 
without hesitation. The bishop failed to confuse her and at 
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last condemned her to death for heresy, bidding her recant if 
she would live. She refused and was lead to prison, from there 
to death. 

While the flames were writhing around her she bade the old 
bishop who stood by her to move away or he would be injured. 
Her last thought was of others and De Quincy says, that re- 
cant was no more in her mind than on her lips. She died as 
she lived, with a prayer on her lips and listening to the voices 
that had whispered to her so often. 

The heroism of Joan of Arc was wonderful. We do not 
know what form her great patriotism took or how far it really 
led her. She spoke of hearing voices and of seeing visions. We 
only know that she resolved to save her country, knowing 
though she did so, it would cost her her life. Yet she never 
hesitated. She was uneducated save for the lessons taught her 
by nature. Yet she led armies and crowned the dauphin, king 
of France. She was only a girl, yet she could silence a great 
bishop by words that came from her heart and from her faith. 
She was only a woman, yet she could die as bravely as any 
martyr who had gone before. 

Sample 220. 

Going Down with Victory. 

I sat on the top of a mail-coach in Lombard street impatiently 
atwaiting the start. 'Twas the night of the victory and we 
would help spread the news over England. 

Up jumps the coachman followed by the guard, an instant's 
preparation, a touch of the lash and we are off! We are soon 
past the limits of the city out in open^ountry, galloping, tear- 
ing along, a clear road ahead of us for the English Mail stops 
for nothing. 

We dash in at villages, stopping but a moment with the mail, 
shouting the news of the victory and we are off again. Proud 
were we and had we not a right to be? The first to carry the 
great news through the land! 

The memory of that ride is ever fresh in my mind and I will 
ever remember those hours as the most glorious in all my life. 

Sample 488. Artificial. 

Letter. 

Your letter gave me real and heartfelt pleasure, mingled with 
no small share of astonishment. Mary had previously in- 
formed me of your departure for London, and I had not ven- 
tured to calculate on any communication from you while sur- 
rounded by the splendours and novelties of that great city, 
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which has been called the mercantile metropolis of Europe. 
Judging from human nature, I thought that a little country girl, 
for the first time in a situation so well calculated to excite 
curiosity and to distract attention, would lose all remembrance 
for a time at least, of distant and familiar objects, and give her- 
self up entirely to the facination of those scenes which were 
then presented to her view. Your kind, interesting and most 
welcome epistle showed me, however, that I had been both 
mistaken and uncharitable in these suppositions. I was greatly 
amused at the tone of nonchalance which you assumed while 
treating of London and its wonders. Did you not feel awed while 
gazing at St. Paul's and Westminister Abbey? Had you no feel- 
ings of intense and ardent interest when in St. James you saw 
the palace where so many of England's kings have held their 
courts, and beheld the representations of their persons on the 
walls? You should not be too much afraid of appearing coun- 
try-bred ; the magnificance of London has drawn exclamations 
of astonishment from travelled men, inexperienced in the world, 
its wonders and beauties. Have you yet seen anything of the 
great personages whom the sitting of Parliment now detains in 
London — the Duke of Wellington, Sir Robert Peel, Earl Grey, 
Mr. Stanley, Mr. O'Connell ? If I were you, I would not be too 
anxious to spend my time in reading whilst in town. Make use 
of your own eyes for the purpose of observation now, and, for 
a time at least, lay aside the spectacles with which authors 
would furnish us. 

Sample 571. 

Venus of Melos. 
In looking at this statue we think, not of wisdom, or power, 
or force, but just beauty. She stands resting the weight of 
her body on one foot, and advancing the other (left) with knee 
bent. The posture causes the figure to sway slightly to one side, 
describing a fine curved line. The lower limbs are draped but 
the upper part of the body is uncovered. (The unfortunate 
loss of the statute's arms prevents a positive knowledge of its 
original attitude.) The eyes are partly closed, having some- 
thing of a dreamy langour. The nose is perfectly cut, the 
mouth and chin are moulded in adorable curves. Yet to say 
that every feature is of faultless perfection is but cold praise. 
No analysis can convey the sense of her peerless beauty. 

More than one hundred of the sets consisting of twenty-seven 
samples were mailed to individuals whose positions as teachers, 
authors, and literary workers implied that they were competent 
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judges of English writing. The following directions accom- 
panied each set : " Please arrange these samples in order of 
merit as specimens of English composition by young people. 
After determining your arrangement, pile all the specimens so 
that the best is on the bottom and the poorest on the top, and 
securely fasten them. Do not consult with anyone." 

When seventy-five replies had been received the results were 
tabulated in the same manner as in the case of the first set. 
Tables VI and VII give the same data for this set as are given 
in Tables III and IV for the first set of judges. Table VIII 
gives the per cent of " better " judgments and Table IX gives 
the differences in quality between the samples. 

Meanwhile Dr. E. L. Thorndike secured the judgments of 
forty-one individuals who were especially competent to judge 
merit in English writing. In order that these results might be 
used as a check on the others, they are tabulated separately in 
Tables X to XIV inclusive. 
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A preliminary examination of the results obtained with the 
set of twenty-seven samples indicated that two other samples 
should probably have been added. These samples, 519 and 520, 
were added and twelve individuals judged this set of twenty- 
nine samples. The distribution of positions assigned the vari- 
ous samples and the number of "better" judgments are given 
in Tables XIV and XV. 

After two hundred and two individuals had judged one or 
the other of the sets of samples, it was decided to form the 
scale. Twenty-one of the samples were common to each of the 
sets. The number of " better '' judgments for these samples 
was obtained by combining the data bearing on these samples 
from Tables IV, VII, XI, and XV. The " better " judgments 
were reduced to per cents and the differences between the 
various samples determined and given in Tables XVI and XVIII 
respectively. The samples composing the scale as given in 
Section 2 were selected as a convenient series from the set 
given in Table XIX. 



50 Measurement of Quality in English Composition 



1 


i 








rHrH 


MCO 


COrH rH 




iH 










CTrH 


rH C<)5D 




00 
00 








rH 


NrH 


tOMCO 




•* 
5 










rHrH N 


(Ncoeo 




fH 

S5 








1^ 


N rH (N 


IN COrH 












^^ 1-^ 


IN(N rH 


T)! rH 




O 
"5 








1-^ 


(NrH(N to 


y-ii~*i^ 




O 








r-i 


U5rH CMrH 


N 




i-H 






^ 


Tfl 


rH rH to 


N 












NrH i*rH 


r-ia rH 






i 








NNi*rH 


N rH 






OS 
f— I 






I-H 


rHCONN 


rH wHy-\ 






CO 






.H (M 


e<i(MrH to 








s 






(N-* 


IN IN 


I-H rH 






05 
"5 






rtCO M 


rHCO rH 








to 






•<J<lNIMrt 


*^i-^fi 












pH 


M t-H rHCO i-t 


rH (N 








s 




Mrt 


■>*»-lrt(N 










00 

to 




p^ »-li-lCO 


NrHrHN 










i 




i-icq N 


T-<nm 










U3 




i-IINlN"5 


i-l rH 










O)' 




MrHlOlM 


i-H ^ 












1-1 


>ocoe<)>-i 












U3 


iH f-H 


lNtO>-l>H 












1—1 

S 


iH ^ni 


(Ni-H 












i-t 

5 


-*««« 




g 


^(Mtjicon 




to 


eoi*co— 1 i-i 




o 
to 


aO(N <N 




StIOII> 


eoj 


rt NCO^^'fl 


tOb-OOOJO 

iH 


1-H rH iH iH rH 


tot»ooo>o 

T^ W-i f-i f~i 0^ 


M<N NINN 


SS5SS 





Results Obtained from Second Set of Samples 51 





1 

g 
1 

•s 

1 
•1 

M 

eg 


f-4 










»-l iH CO f— 1 


TfCOiO 




00 
00 








v-4 


rH Tjl-^CO 


toco 




i 








1-t 


IN-* -^t-US 







^ 
•* 








1H 


IN^INIOtH 















1-1 IN 


Tt<lOt>t~ 






"5 








tH 1— I rH 


IN-* CO 














IN IN-HCN 


IN "5 






I 








i-H t-H 1— ( t— 1 CO 


to 




^ 


1—1 

g3 


(NfN-HCOM 




i 








CO "5 CO CO 








(NWtD 


", 




s 






l-H fH i-H 


COCO 








i 






CO coco 


U3 









■* 
s 


CO ^ CO 


03 
US 


■* -110 


.s 


IN 

to 




CO 


NCDCO 








£ 


s 




10 


00>-l 








^1 






CO 


CO 








R 



to 


■*to 


J 


00 
1— t 

to 


rH U5 


^ 


U5 


■* 




g 





toa> 












g 





n 












S 


^rt (MO 














CD 


CO 0000 




1— 1 

s 


C^(Nrt 




§ 


CO«D 




1— 1 


(N 




pajBdx 


noo 


l>.-IOOi-l 
05Ui t* 

to-* lOtO 


005 OS U5 00 

10 10 to 


coo-*t^o> 

Ot-03INiH 
tO"5 tOUJ 


COINOOlIN 


•HOO"-! 

IN IN t-^ IN 
<N CO U5 10 C-l 


■*|'I5 00 
CO to 00 
^10 •* 



52 Measurement of Quality in English Composition 



a 



^ 



» 

o I* 

(H 

n I— ( 
1^ 

9 B 
- S 2 

" a 

o 
v 

>. 

J3 



I 



a iJ 
pq 



p. 


rH 






N(N 


CO lOCOOO 


§s 


00 
00 






^5 


^ gss 


§ 


"5 






s§ 


CO l>tOOS 




■* 

s 






eoio 


05 00 I- "5 

eoiot^t- 




i 






t^usoo 

C0"5O 


CO o 




U5 








o 




■X- 






ooo 


^ 




i 




1-» 
IN 


mS°Sot 






I— ( 




s 


OIXMO 

"i<COU5 00 






?3 






-Hooo 
to CO to 






1— ( 




?5SS 


ooo 






8 




2J3K 


s 






CO 


OIM ■* 

rHIN 00 


"5 




» 

1— t 




s 


?5?3 


Sg 








05 




<N 

O 








1 


""^S 


i 


"'gg 


00 

«3 


1-1 00^ 




^gj 


"5 


CO 


9[dm 


noa 

5S 


t-omooeo 


* 

t-OlC^.-HC0 
us tO>OU5 


NOtOCOiH 
MNi-lcON 


0(N t^NCO 
CO us ION-* 


U5 00 



Tl 




s 












GQ 




m 








bl 




T) 




:3 








(N 








C4 




o 




^ 












o 




o 




00 


^ 


|3 


s 


4J 


.2 


1 


Jh 




,C1 


g 








y^ 


(N 


to 


«4^ 


a 


o 




-M 


Tl 


0) 




V) 




s 


a 


-tJ 


A 


CI 


.ja 






•a 


^ 


1 


s 






g 


C3 




o 






g 


s 




J3 


i) 




^ 


£ 


us 


to 

1— 1 


Tl 


-»a 


a 


a 


cd 


a 



■gS) 

i.a 

e o 

il 

Co C3 

03 § 

* a 
o 



Results Obtained from Second Set of Samples 



> cj 

n 



o 

n 
S 
p 

H 



a 
.i 

1 

S 

c3 
02 








00-* 
-nd 

1— 1 I-H 


17.8 

25.3 
19.3 
39.6 


f-t 1— 1 


1 






COCO 


23.8 

29.2 
29.7 
42.1 


"5 








■*00 

cod 


16.3 

36.6 
30.7 
44.6 




s 

•* 






00 CO 

d«5 

r-lIN 


COt~.-lr-l 




o 

IN 






CO (Nt- 

odt^co 

-IINCO 


33.2 
54.0 




^ 
5 






TtlCOt- 

■-idiN 

rt (NCO 


d 

IN 




« 






t~co 

(NCO 






i 




d 


00-* 00 IN lO 

CD O 03U5 t^ 

^ w ^ CO -<* 






,-4 

(N 




CO 

T— 1 


00 cor- CD 
o>o6»o oi 

rf rt INCO 






CO 
IM 

CO 




CO 


30.2 
18.8 
39.6 










Th ^ CO 


f-H T— 1 






o 




7.9 
15.4 
38.6 


CD 
IN 






IN 
IN 

CO 


5.0 
10.9 

41.6 




CD O! CO 00 


* 

CTl 

l-H 


i-H t~ 


(N 

CO 


r-i CO 
i-H I-H 


t-CD 
IN CO 










8.4 
23.3 
29.7 










o 


4.0 
20.3 
25.3 


CD 


4.5 
18.8 
42.6 


00 

CO 


lOOIN 




OOi 

i-H 


i 


I-H 


9idni 


II03 

88 


t-oioooco 

O00OJ.-1 o 

CO "CO toco 


O-* t-OlTfl 
t^OllNrHCO 
U3 (0>0"3 


(Noeoco-H 
(NOcncqiN 

CO IN .-ICON 


* 

0«t~lNCO 

C010U3N-* 


MS 00 



54 Measurement of Quality in English Composition 



> 

M 

o 

EQ 

m 
o 



PS "-I 
H >^ 

.2 

!^ 

Pi M 
H H 



S' 



O ft, 

Who 






CQ Q 

P CD 



" (0 
IS «^ 



s 



GO 

1 
1 

a 

8 

•s 

& 

■? 
J 


t- 
t- 








CO OiCQ 


§8 






i-H 1-1 


CD I-H OS 

o oor» 


CO 






§8 

(Ni-I 


U5 f-H U3 
tH lot- 








1-1 


oscoiooa 

(NOOtJIi^ 

1-H 


CM 






CO OICO 


CO 








05 Tt< CD 
<-H rH 




» 

o 

"5 


t~co 


1 




I— 1 


1— 1 t— I i-H 




(M 




f-H 

to 


cOrHI>OS 
(NCOOiCO 




CO 
(N 

CO 






l> I-H 05 
l>COCC 

I-H 




CO 

r-t 






g 




O 




C»H-IC0 


SI 




1 

CO 


TfCO (N 

S2 « 




COOC<I r-l 

(NNhH hH 


* 


i-H 


1> 




00 t- 
00 Tf 






s 


C4 T-4 


"5 


1 §s§ 

N r-l 


CO 

g 


lOMiN 


00 

iH 


Il- 




ls 




1 


8[dni 


noo 


t-0"5 00CO 
OOOOJrHO 
ZDUiUitDtO 


* 

t-OJM.-IC0 
>0 ffl"5U5 


(NOOCO.-I 
(N005(N(N 
COIN-HCOM 


* 

OOi-HO-* lOOO 

ON t— N CO eooo 



Results Obtained from Second Set of Samples 55 



TABLE XIX 

The Samples Selected for the Scale, the Dippbrences Between 

Adjacent Samples and the Difference Between Each Sample 

AND Sample 580 for Each of Two Groups op Judges 



Sample 


Group of 202 judges 


Group of 41 judges pi*-^ 


Dififerences each 
from preceding 


Differences 

from sample 

580 


Differences each 
from preceding 


Differences 

from sample 

580 


580 
595 
618 
94 
519 
534 
196 
221 
571 
177 


183 

77 

109 

105 

111 

90 

97 

66 

99 


000 
183 
260 
369 
474 
585 
675 
772 
838 
937 


173 

81 
141 

* 

246t 
91 
91 
41 
71 


000 
173 
254 
395 

641 
732 
823 
864 
935 



JL^ 



* This sample is not in the set judged by this group, 
t This is the difference between samples 94 and 534. 



Section 9. Sufficiency of Judges 

No claim is made that the values given in the scale are ab- 
solutely accurate. Variation among the judges was very great, 
and to make a perfect scale would require the services of many 
more judges than it was possible to secure for this study. The 
scale is accurate enough to be of very great practical value in 
measuring the merit of English compositions written in the 
upper grades of the elementary school and in the high school. 
The scale will also serve as the basis of future efforts in this 
direction, and it can be refined and perfected part by part. 

The scale was based on the judgment of two hundred and 
two individuals. Several checks were employed to test the 
accuracy of the work done by these judges. Table XIX shows 
the differences between adjacent samples and the absolute values 
for the samples of the scale. One part of the table shows the 
results for all the judges and the other part shows the results 
obtained from the group of judges described on page 40. The 
differences in the absolute values in the two results are surpris- 
ingly small. In no case do the values vary by a complete step 
in the scale, and the value of the best sample is almost the same 
in both results. This similarity of values indicates that the 
values in the scale are reasonably near the values that experts 
in English composition would assign to the samples. 

If the scale were accurate, the values of the samples would 
remain the same no matter how they were obtained from Table 
XVIII. Thus the difference in value between samples 580 and 
618 in the scale is (183-1-77) 260, and in this case the table 
shows the same difference when sample 580 is compared directly 
with sample 618. The agreements that exist between the results 
obtained by these methods are seen in the following tabulation: 

Difference between samples 580 and 618 in scale is 260, in Table XVIII 260 

« " 595 " 94 " 186 " 204 

94 " 534 " 226 " 209 

534 " 221 " 187 " 161 

'• " 196 " 571 " 163 " 179 

« « " 221 " 177 " 165 " 186 

56 



Section lo. The Quality of Judges 
The question of what individual or group of individuals rep- 
resents the best judges of English composition is a difficult one 
to answer. The data at hand do not show any marked dif- 
ferences between the various sets of judges so far as uniformity 
of judgment is concerned or faithfulness to the proper order of 
the samples. In order that some definite conclusion might be 
reached concerning the reliability of judges, the following scale 
of penalties for placing samples in the wrong order was de- 
veloped. Where the diflferences between samples are very great 
the penalty for misplacing has been made correspondingly large. 
Thus the difference between samples 627 and 534 is 129. If 
the judge considers sample 534 better than sample 627, he is 
guilty of a much greater error than he would be in placing 
sample 571 before sample 220, where the difference is only 15. 
In the latter case the penalty was arbitrarily fixed at i while 
in the former it was placed at 6. 



TABLE XX 

The Differences Between Cbktain Sample Compositions and the 

Scale of Penalties for Placing Samples in Wrong 

Order by Individual Judges 



Sampl 


es 


Ascertained 


Penalty for 


compared 


differences 


wrong order 


94 with 


627 


47 


2 


627 


" 


534 


129 


6 


534 


u 


322 


32 


2 


322 


It 


200 


43 


2 


200 


u 


196 


20 


1 


196 


u 


323 


66 


3 


323 


t( 


221 


39 


2 


221 


tl 


300 


10 


1 


300 


u 


220 


64 


3 


220 


it 


571 


15 


1 


571 


tl 


434 


45 


2 


434 


it 


565 


20 


1 


565 


it 


488 


5 





488 


K 


177 


22 


1 
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The following tables show the penalty each judge in each 
group received and also the median penalty for each group. 



TABLE XXI 

The Penalty for Wrong Order Received by Each or the 

Individuals in the Several Groups or Judges 



Group of 74 


Group of 75 


Special Group of 41* 


t 

>-5 


1 


|-» 


1 
1 


1 

1-5 


1 

a 






1 

1-5 


>> 
1 

(2: 


1 


5 


39 


8 


1 


7 


39 


8 


1 


7 


2 


5 


40 


6 


2 


6 


40 


7 


2 


13 


3 


6 


41 


8 


3 


11 


41 


9 


3 


10 


4 


19 


42 


11 


4 


5 


42 


5 


4 


13 


5 


7 


43 


17 


5 


10 


43 


10 


5 


8 


6 


9 


44 


5 


6 


12 


44 


6 


6 


8 


7 


10 


45 


10 


7 


9 


45 


9 


7 


8 


8 


8 


46 


7 


8 


10 


46 


11 


8 


6 


9 


5 


47 


9 


9 


9 


47 


9 


9 


10 


10 


12 


48 


12 


10 


12 


48 


6 


10 


13 


11 


6 


49 


8 


11 


15 


49 


9 


11 


9 


12 


15 


50 


8 


12 


11 


50 


9 


12 


16 


13 


10 


51 


8 


13 


8 


51 


9 


13 


5 


14 


10 


52 


11 


14 


5 


52 


4 


14 


14 


15 


7 


53 


11 


15 


10 


53 


9 


15 


9 


16 


9 


54 


10 


16 


10 


54 


14 


16 


11 


17 


9 


55 


6 


17 


16 


55 


3 


17 


2 


18 


5 


56 


7 


18 


11 


56 


11 


18 


13 


19 


11 


57 


12 


19 


14 


57 


13 


19 


12 


20 


8 


58 


10 


20 


9 


58 


7 


20 


10 


21 


12 


59 


15 


21 


14 


59 


7 


21 


12 


22 


7 


60 


12 


22 


8 


60 


10 


22 


5 


23 


9 


61 


12 


23 


3 


61 


11 


23 


7 


24 


14 


62 


12 


24 


9 


62 


6 


24 


9 


25 


4 


63 


10 


25 


16 


63 


5 


25 


11 


26 


4 


64 


6 


26 


16 


64 


8 


26 


11 


27 


12 


65 


7 


27 


10 


65 


13 


27 


7 


28 


14 


66 


7 


28 


13 


66 


4 


28 


6 


29 


9 


67 


11 


29 


13 


67 


8 


29 


5 


30 


4 


68 


11 


30 


8 


68 


11 


30 


13 


31 


15 


69 


12 


31 


9 


69 


14 


31 


5 


32 


12 


70 


6 


32 


8 


70 


15 


32 


6 


33 


7 


71 


8 


33 


5 


71 


8 


33 


2 


34 


10 


72 


10 


34 


7 


72 


9 


34 


7 


35 


11 


73 


10 


35 


6 


73 


13 


35 


11 


36 


6 


74 


11 


36 


5 


74 


15 


36 


14 


37 


17 


75 




37 


3 


75 


8 


37 


6 


38 


7 






38 


9 






38 





* Data were not calculated for four of the judges in this group. 
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TABLE XXII 

Distribution op Penalties Given in Table XXI 

The table reads as follows: In the group of 41 each of two judges 
received the minimum penalty, 2; in the group of 75 each of three judges 
received a penalty of 3 ; in the group of 74 each of three judges received 
a penalty of 4, etc. 



Penalty 


Group of 74 


Group of 75 


Special 
group of 41 


2 






2 


3 




3 




4 


3 


2 




6 


4 


6 


4 


6 


7 


5 


4 


7 


9 


5 


4 


8 


9 


7 


3 


9 


6 


12 


3 


10 


10 


7 


3 


11 


8 


9 


4 


12 


10 


3 


2 


13 




5 


5 


14 


2 


4 


2 


15 


3 


3 




16 




3 


1 


17 


2 






18 








19 


1 






Median 


9.8 


9.8 


9.6 



Table XXII shows the distribution of penalties received by 
each group of judges. It will be recalled that the group of 74 
represents the individuals who judged the set composed of 
83 samples. These persons were largely graduate students of 
education. The group of 75 is composed of those who were 
selected for their general ability as competent judges of Eng- 
lish composition. The special group of forty-one is composed 
of those who may be said to be expert judges of English com- 
position. The table of distribution of penalties shows no im- 
portant differences in the three groups. The medians for the first 
two groups are 9.8 in each case while the median for the last is 9.6. 

By referring to Table XXI, the individuals whose penalties 
are at the extremes of the distributions may be found and 
from data in the possession of the author their identity may be 
determined. In the special group of 41, judge number 17 re- 
ceived a penalty of 2. This individual is a psychologist, a clear, 
convincing writer and an editor of large experience. Were his 
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name mentioned here it would be that of a man who has an 
international reputation. The largest penalty received by any 
judge in this group was i6 by judge number 12. All that was 
said regarding the ability of judge number 17 could be said of 
this judge, except that his field is philosophy instead of psy- 
chology, and he has not had a wide experience in editing. These 
individuals would be considered equally competent as judges of 
English composition, yet their judgments show the widest diver- 
gence found in this group. Judge number 23 in the group of 75 
has a penalty of 3. This individual is a woman who reads a 
great deal of the best literature. She is not a teacher, and ex- 
cept for occasional papers prepared for a study club, she does 
not write. Judge number 17 in this same group has a penalty 
of 16. This individual is the author of several novels and 
descriptive works. He has had considerable experience as a 
newspaper reporter and was for some time the editor of a 
paper. Judge number 6 in the group of 41 is the author of 
several texts largely used in the English work of the high school. 

Section ii. The Use of Mixed Types of Composition 

Two methods of selecting samples from which a scale might 
be derived are possible. The first is to take only compositions 
that would be classified as narration or as description. The 
second method is to pay no attention to such distinctions ; rather 
to make the effort to include all the various types. Either 
method is good. In this study the effort was made to include 
all types except poetry. It was believed that in this way the 
scale could be made of more value to the teacher. In the upper 
elementary grades and in the secondary schools the distinction 
between narrative, description, and argument is not an impor- 
tant one. Thus the scale will measure the actual products ob- 
tained in these grades. The objection may be offered that it 
is impossible for a person to compare a sample of narration 
with a sample of description, that we are trying to compare 
entirely different things. The answer is that people actually did 
do it. Of the four hundred and fifty people who have judged 
these samples not more than three have offered any objection 
on the score that they could not compare the samples. 



